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Abstract. Information system designers face many challenges w.r.t. se- 
lecting appropriate semantic technologies and deciding on a modelling 
approach for their system. However, there is no clear methodology yet 
to evaluate "semantically enriched" information systems. In this paper 
we present a case study on different modelling approaches for annotat- 
ing medical images and introduce a conceptual framework that can be 
used to analyse the fitness of information systems and help designers to 
spot the strengths and weaknesses of various modelling approaches as 
well as managing trade-offs between modelling effort and their potential 
benefits. 



1 Introduction 

Information systems can have very different shapes and variants and designing 
such systems involves taking important modelling decisions to optimise informa- 
tion retrieval. The different dimensions to be taken into account are performance, 
maintainability and usability of the system. Some research has been carried out 
to evaluate information systems w.r.t. these dimensions [3l4l6j . however, none 
of these approaches analyses the quality of such a system w.r.t. its queriability, 
i.e. how easy or comfortable it is for a user to formulate queries. Since informa- 
tion retrieval is the main purpose of an information system, we believe that the 
queriability and therefore the difficulty of assessing the information is a crucial 
point. 

Recently, there has been a lot of discussion about "semantically enriched" 
information systems, especially about using ontologies for modelling data. On- 
tologies have the potential of modelling information in a way that they can 
capture the "meaning" of the content by using expressive knowledge represen- 
tation formalisms (such as Description Logics pQ) and therefore achieve good 
information retrieval results. However, ontology-based information systems can 
come in many different fashions. It has to be decided how expressive the con- 
ceptual schema is, whether it is possible to use an off-the-shelf schema or one 



that is tailored to the application. Furthermore, it is crucial that the schema 
and the data are well-suited in order to enable good queriability and retrieval 
performance. Depending on the design decisions, more or less modelling effort 
is involved. Information system designers need a methodology that helps them 
to understand the benefits and trade-offs of various approaches and to make an 
informed decision about the "optimal" modelling approach for their needs. 

In this paper we present a case study on medical image annotations. Medical 
images are usually stored in large databases or file systems along with various 
information about the images, such as radiologists reports (usually formulated 
in natural language). Although it is possible to retrieve images based on these 
descriptions with full-text search and keyword queries, the results are prone to 
have a low recall and precision [8] . There is a wide range of alternative modelling 
approaches. The relevant terms in the natural language descriptions could be 
mapped to an underlying schema, such as a thesaurus or to an ontology. The 
more expressive the schema, the more "meaning" can be potentially modelled 
in the image annotations. This might result in better retrieval performance but 
also in more modelling effort while creating the annotations, e.g. depending on 
whether or not the information extraction can be done automatically or has to 
be done manually. 

There is a trade-off between how much effort is involved in the design of the 
system and the creation of meaningful image annotations and how this leads 
to better queriability and better quality of the retrieval results. In order to 
understand this trade-off and to analyse and compare different approaches for 
modelling medical image annotations we will use a conceptual framework [5] that 
has been designed for the evaluation of information systems with a particular 
focus on queriability. We will outline five modelling approaches for medical image 
annotations that range from a simple text-based information system to full- 
fledged ontology-based information systems based on an established medical 
ontolog y, namely SNOMED CT0 Applying the framework to these modelling 
approaches will highlight their strengths and weaknesses and the framework's 
measurements will allow us to compare the modelling approaches with each 
other. This case study is one of several studies that we are carrying out to 
evaluate and refine the framework. 

The remainder of the paper is organised as follows. In Section [2] we outline 
several possible modelling approaches for medical image annotations. Section 
[3] introduces the Evaluation Framework and how it can be applied to measure 
the fitness of a modelling approach. In Section 0] we apply the framework to the 
particular modelling approaches that we outlined for medical image annotations. 



SNOMED CT: |http : //www . ihtsdo . or g/snomed - ct/ 1 



We measure and compare their fitness and queriability and discuss the weak and 
strong points of each modelling approach. Section [5] concludes the paper. 



2 Modelling Approaches for a Medical Image Information 
System 

An information system for medical image annotations can be designed in many 
different ways. In the case study described in this paper we distinguish between 
three general categories of modelling approaches. A simple text-based informa- 
tion system, a slightly more advances thesaurus-based information system and an 
ontology-based information system. For the latter we distinguish between three 
different ways of designing an ontology-based information system, differing in 
the expressivity of the underlying schema and the annotations. 

All of the five modelling approaches are based on the same data corpus. We 
used 42 publically available chest radiology images and their natural language 
(English) radiology reports from the web-based radiology database EURORADo 
The reports contain information about image type and modality, findings, body 
parts, diagnoses, etc. An early experiment on image retrieval with this data set 
has been published in jS]. The original data, i.e. the natural language reports, 
are processed differently for each of the five modelling approaches. 

The various approaches described below are representative. Obviously, there 
are other alternatives or mixtures between those mentioned. Some of them could 
be partly improved with some customising. The selection in this paper represents 
distinct groups of modelling approaches that we chose deliberately to highlight 
their strengths and weaknesses. 

2.1 Text-based Modelling Approach 

In a text-based modelling approach (MA\) for our medical image information 
system the data collection consists merely of the unprocessed natural language 
descriptions. The queries are conjunctions of natural language keywords and 
the query results are obtained by carrying out a full-text search over the image 
descriptions. 

This method involves very little modelling effort and simple querying (like 
in an internet search engine). However, the approach is not very powerful: nei- 
ther the queries nor the underlying data can identify synonyms, homonyms, 
acronyms, spelling mistakes or capture taxonomical or relational information. 
Therefore, a text-based approach is not suitable for capturing the full semantics 
of the image descriptions and most likely leads to low recall and precision of the 
retrieval results. 

2 EURO RAD : |http : //eurorad . org/] 



2.2 Thesaurus and Text Mining based Modelling Approach 



For this modelling approach (MA 2 ) we processed the natural language descrip- 
tions with publically available, off-the-shelf text mining tools. We used GENIA 
taggeid to extract noun phrases, verb phrases and adjective phrases from the 
textual descriptions and processed these with MetaMapQ in order to map them 
to SNOMED CT classes via UMLsl For each image an annotation file was 
created containing a list of SNOMED CT classes that reflect the relevant terms 
extracted from the text. Although SNOMED CT is strictly speaking an ontology 
that can be expressed in OWL and conforms to the OWL 2 profile OWL EL 
[7] , we merely use it in the sense of a thesaurus for this modelling approach, i.e. 
each image is "tagged" with a list of SNOMED CT classes and we only make 
use of the synonyms and the taxonomical information that are contained in the 
ontology and do not capture any relational information. 

Again, the queries are expressed in natural language and are processed in 
the same way and with the same tools as the image descriptions. Each query is 
transformed to a list of SNOMED CT classes and matched against the image 
annotations. 

The advantages of a thesaurus-based modelling approach compared to a 
purely text-based approach arc that recall and precision can be increased due to 
the fact that a thesaurus can recognise synonyms to a certain extent and contains 
taxonomical information so that image annotations that contain subclasses of 
the query terms are retrieved as well [5] . The additional modelling effort required 
to achieve these benefits is limited to incorporating an off-the-shelf thesaurus and 
processing the textual descriptions automatically with off-the-shelf text mining 
tools. On the other hand, this approach does not capture relational information 
and the tools do not map all the available information perfectly, i.e. the trans- 
lation process is error prone (e.g. mapping to wrong class or no mapping at all, 
inability to recognise acronyms). 

2.3 Ontology-based Modelling Approaches 

All of the three ontology-based modelling approaches we discuss in this paper 
(MA3, MA4 and MA$) involve translating the textual image descriptions manu- 
ally to Abox assertions to different schemas (SNOMED CT Tboxes). The queries 
are formulated as OWL class expressions. 

We distinguish between three variants of ontology-based annotations: 



3 GENIA tagger: 
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4 MetaMap: http://metamap.nlm.nih.gov/ 

5 UMLS: http://www.nlm.nih.gov/research/umls/ 



— MA^: image annotation Abox (Ai) with class assertions to a SNOMED CT 
Tbox (Ti) 

— MA4: image annotation Abox (A2) with class and role assertions to a SNOMED 
CT Tbox (Ti) 

— MA5: image annotation Abox (A3) with class and role assertions to a SNOMED 
CT Tbox (T2) that contains some additional "image- annotation-specific" 
roles 

MA5 uses a slightly different Tbox than MA3 and MA4 in the sense that we 
created an additional set of roles and a role hierarchy in order to bypass the 
SNOMED CT specific role groups. An example of a disease in SNOMED CT 
that is defined using role groups is NeoplasmOfLung. The concept is defined as 
follows!! 



NeoplasmOfLung = DisorderOfLung n 

3roleGroup( 3AssociatedMorphology. Neoplasm n 
3FindingSite.LungStructure) 

For MA5, we introduced three additional roles: shows, hasFinding and hasLo- 
cation and defined the following role hierarchy: 

roleGroup o AssociatedMorphology C hasFinding 
roleGroup o FindingSite C hasLocation 
shows o hasFinding C shows 
shows o hasLocation C shows 

If we want to find all images that show neoplasms in MA& , we can formulate 
a simple OWL class expression query like Image nBshows. Neoplasm and would re- 
trieve images labelled with ImagenBroleGroup.BAssociatedMorphology. NeoplasmOfLung 
without having to use the complicated role group construct in the query. 

Furthermore, we introduced the role derivingFrom in order to indicate a 
causal relationship between two findings, e.g. a metastasis and a primary tu- 
mour. 



Similar to the thesaurus-based modelling approach (see Section 1212]) we use 
an off-the-shelf schema. By using an underlying ontology we can take advantage 
of all the benefits that a thesaurus-based modelling approach involves (e.g. syn- 
onyms and taxonomical relations between classes). MA4 and MAq additionally 
capture relational information and are therefore better suited to capture the ac- 
tual meaning of the image descriptions and increase recall and precision of the 
results compared to the other modelling approaches. 

6 To improve readability, we use slightly abbreviated class names and DL syntax. 



On the other hand, all of these modelling approaches involve a high modelling 
effort due to the manual translation of natural language to ontology-based an- 
notations. This requires both domain knowledge as well as knowledge of OWL. 
Furthermore, the benefits that these approaches potentially implicate can only 
be taken advantage of if the schema, data and queries are well-suited. 

3 Information System Evaluation Framework 

We start by formalising the relevant components of a (semantically enriched) 
information system for which we are then going to evaluate and compare the 
different modelling approaches. We will use the term "modelling approach" to 
describe the whole system consisting of data, schema, (an abstraction of) queries, 
and a query language. A more detailed description of this framework can be 
found in [5]. 

3.1 A Modelling Approach 

A modelling approach MA = (S, D, R,QL) consists of 

— a schema S: a finite description of the semantics of the data, e.g. a database 
schema, a logic program, or the Tbox of an ontology, which can be empty. 

— the data D: e.g. tables and rows in a relational database, ground facts, or 
ontology Abox assertions. 

— a set of information requests R: each r G R represents the answer to a 
query of D, and is given as a set (of tuples) over D. Ideally, R should be 
representative for the queries to be answered by the information system to 
be built. 

— a query language QL: e.g. SQL, (union of) conjunctive queries, OWL class 
expressions. 

An information request asks for tuples of the given data that are relevant for 
the user. The request needs to be distinguished from the actual query, which is a 
specific manifestation of the information request formulated in QL. An informa- 
tion request r can correspond to 0, 1 or more queries in a given query language. 
The former is the case if there are no queries in QL whose answers would be 
exactly the tuples in r when asked over S and D, i.e., if QL is unable to express 
the information request over the given schema and data. In the case that there 
are one or more queries, some of them might be more easily expressible than 
others. 

The only assumptions we make is that the query language QL comes with a 
semantics that identifies, for a given query q of arity n in QL, data D, and schema 
S, the set of certain answers [2J. More precisely, we assume the existence of an 



entailment relation [=, and use Ind(D) for the set of individuals or constants in 
D to define cert(-) as follows: 

cert(q, S, D) — {w e Ind(D) n \ S U D |= g(w>)}. 

3.2 Measuring the Fitness of a Modelling Approach 

The basic characteristic we want to evaluate is the fitness of a modelling ap- 
proach, i.e. how well the schema and the data are suited to enable the formu- 
lation of "fit" queries for answering the given information requests. The fitness 
of a modelling approach can be determined by analysing the syntactic, semantic 
and/or cognitive complexity of the queries that correspond to the information 
requests and depends on the fitness function. 

The Fitness Function Different queries that correspond to an information 
request can vary in length and be more or less complex, e.g. in terms of using 
relations and constructors such as conjunctions, disjunctions, etc. They can also 
be more or less difficult to understand from a cognitive perspective. For example, 
a human user might find a query that uses terms that are actual words (in the 
sense that they exist in a domain expert's dictionary) easier to understand than 
one that uses anonymous identifiers. The purpose of the fitness function is to 
capture this complexity. 

The framework is parameteriscd with a fitness function f that associates each 
query q in QL with some value f(q) that is intended to capture its fitness. We 
only require that / maps QL into a totally ordered set (M, <), e.g. K or N 4 , which 
we call the query's fitness value. Obvious examples of fitness functions are (i) a 
query's length, (ii) a query's length combined with the number of constructors 
involved, either via some (weighted) summation or into a vector, or (hi) a query's 
length combined with the number of terms not to be found in a domain expert's 
diactionary, or any combinations or extensions of these. 

The smaller the fitness value, the "better" the query. We read f{q) < f(q') as 
q being "better" or "fitter" than q'. The framework evaluates the "best queries" 
for an information request, e.g., the shortest and least complex queries. The 
fitness function induces a partial order on the queries. 

The Query Space Each information request r 6 fi has an associated query 
space: first, we define correct queries cQ(r, S, D) as those that answer exactly 
an information request r over S and D: 



cQ(r, S,D) = {q | q is a QL query and cert(q, S, D) = r(D)}. 



Next, we define best queries bQ{r, S, D, f) as those correct queries whose 
fitness is maximal. Clearly, best queries depend on how we measure fitness, and 
thus on the fitness function /: 

bQ(r, S, D, f) = {qe eQ(r, S, D) j there is no q G cQ(r, S, D) : 

/(</) < /(<?)}■ 

Since the bQ(-) are the "fittest" queries among the correct queries, any two 
queries in bQ(-) are equally fit, and we can abbreviate their fitness as follows: 
for f(q t ) = f(qj), we set f{{q x , ...,q k }) to be f(q x ). 

3.3 Using the Evaluation Framework to Compare Modelling 
Approaches 

If we want to compare several modelling approaches, we can compare, for each 
information request rj S R and each of the modelling approaches, the fitness 
of the best queries. This can unveil the strengths and weaknesses of the infor- 
mation system to the system designer. For example, if there are information 
requests for which the set of correct queries is empty, then f(bQ(r, S 1 , D, /)) is 
prohibitively bad. To overcome this, we can then decide whether to select a dif- 
ferent, more powerful query language or to change the schema or the way the 
data is modelled — or whether perhaps that particular information request is of 
too little importance for such a change. The measurements can also help to point 
out where the trade-offs between modelling effort and benefits in terms of eas- 
ier query answering are. For example, considering an ontology-based modelling 
approach, whether more modelling effort for a more expressive Tbox would be 
justified for the sake of simpler queries. 

Applying the framework to one modelling approach MA = (S, D, R, QL) re- 
veals for each r, the fitness value of the best queries: f(bQj). In particular, it 
will identify information requests for which it is hard to specify a query in QL 
and those for which this is impossible. When comparing different modelling ap- 
proaches we can compare the point-to-point fitness for each information request. 

4 Applying the Framework to a Medical Image 
Annotation System 

We will now demonstrate how the evaluation framework can be applied to the 
various modelling approaches we sketched in Section [5J The goal of this evalua- 
tion is to find out how to model the information in order to get optimal retrieval 
results, how much effort is involved in this "optimal" modelling and how fit the 
modelling approaches are w.r.t. queriability. 



4.1 Information Requests 

A set of representative information requests R is derived from the content of 
the original, natural language image descriptions: image types and modalities, 
clinical findings, complex findings (e.g. involving locations) and combinations of 
the former. 

— r\: involves one clinical finding: "All images that show neoplasms." 

— r 2 : involves two concepts, an image type and an image projection: "All X-ray 
images with lateral projection." 

— r 3 : involves a clinical finding combined with a qualifier value: "All images 
that show left-sided pleural effusions." 

— involves a clinical finding combined with a body structure: "All images 
that show soft tissue masses in the pleural membrane." 

— r§: involves a causal relationship between two findings: "All images with 
metastases deriving from a carcinoma." 

The information requests are extensional, i.e. they are mapped to a set of 
answers from the data space. Above, we listed English descriptions of n to r§ 
for better understanding. 

4.2 Modelling Approaches 

We formalise the modelling approaches that we introduced in Section [5] 

— MA\ = (0, text, R, keywords) 

— MA<x = (Ti, concept list, R, keywords) 



Note that all representation of the data are derived from the same data set, 
i.e. the original natural language image descriptions. In the various modelling 
approaches we tried to extract information from this corpus in different ways, 
some of which capture more "meaning" than others. 

4.3 Results 

Tabic 1 and Table 2 show for each of the above mentioned information requests 
and modelling approaches both the best query and its fitness vector of the form 
m = (x, y, z) where x is the length of the query, y is the number of distinct con- 
structors and z indicates the nesting depth. For better readability of the queries 
we abbreviated some of SNOMED CT's role names: AM stands for Associat- 
edMorphology, FS stands for FindingSite, QV stands for QualifierValue and rG 
stands for roleGroup. 



MA 3 
MA 4 
MA 5 



(Ti.Ai, R,CL) 
(T 1 ,A 2 ,i?,GL) 
(T 2 ,A 3 ,i?,GL) 



r 3 


MAi 




AM 3 


T\ 


neoplasm 


Neoplasm 


Image l~l 

3shows. Neoplasm 




mn = (1, 0, 0) 


mi2 = (1, 0, 0) 


mia = (3,2, 1) 


T2 


xray l~i 
lateral 


Xray n 

LatcralProjcction 


Image l~l 
3shows.Xray l~l 
3shows. LatcralProjcction 




m 2 i = (2, 1,0) 


m 22 = (2, 1, 0) 


m 23 - (5,2,1) 


r3 


pleural effusion n 
left 


PleuralEffusion l~l 
LeftSidcd 


Image l~l 

3shows. PleuralEffusion n 
3shows. LeftSidcd 




77131 = (2, 1,0) 


77132 = (2, 1,0) 


m 33 = (5,2, 1) 


r 4 


soft tissue mass l~l 
pleural membrane 


SoftTissueMass n 
PlcuralMembrane 


Image n 

3shows. SoftTissueMass n 
3shows. PlcuralMembrane 




77141 = (2, 1, 0) 


m 42 = (2, 1,0) 


77143 - (5,2, 1) 




metastasis n 
carcinoma 


Metastasis n 
Carcinoma 


Image n 

3shows. Metastasis l~l 
3shows. Carcinoma 




77151 = (2, 1,0) 


m 52 = (2, 1,0) 


77153 = (5,2, 1) 



Table 1. Measurements for MA\, MA 2 and MA 3 . 



r 3 


A/A 4 


MA 5 


n 


Image n 3shows. (Disease n 
3rG. (3AM. Neoplasm)) 


Image l~l 
3shows. Neoplasm 




mi4 = (6, 2, 3) 


77115 = (3,2,1) 


V2 


Image n 

3hasImagcTypc.Xray l~l 

3 hasImagcProjcct ion. LatcralProjcction 


Image n 

3hasImagcTypc.Xray l~l 

3 hasImagcProjcct ion. LatcralProjcction 




77124 - (5,2,1) 


77125 = (5,2, 1) 


r 3 


Image n 
3shows. (PleuralEffusion l~l 
3QV. LeftSidcd) 


Image n 
3shows. (PleuralEffusion n 
3QV. LeftSidcd) 




77134 - (5,2,2) 


77135 - (5,2,2) 


r 4 


Image f~l 3shows. (Disease l~l 3rG.( 
3AM. SoftTissueMass n 
3FS .PleuralMembranc) ) 


Image n 

3shows. (3hasFinding. SoftTissueMass n 
3hasLocat ion. PlcuralMembrane) 




77144 = (8, 2, 3) 


77145 = (7,2,2) 




Image l~l 
3 shows. Metastasis n 

3shows. (Disease n 3rG. (3AM. Carcinoma)) 


Image l~l 
3shows. (Metastasis n 
3dcrivingFrom. Carcinoma) 




m 54 = (8, 2, 3) 


77155 = (5,2,2) 



Table 2. Measurements for MA A and MA 5 . 



4.4 Analysis 



The first observation we make is that not all of the queries noted in the tables 
return the correct answers to their corresponding information request. A query 
on a gray background indicates that it answers the information request only 
partially. In these cases the modelling approach is not expressive enough to 
capture the full meaning of the image description. For example, the queries for 
MA\ are merely keywords, i.e. they cannot capture synonyms, homonyms etc. In 
r\ we ask for image descriptions that contain the word "neoplasm" . If we have a 
description that contains the synonym "tumour" instead, it will not be retrieved 
and the results therefore have a low recall. In we ask for "Xray images with 
lateral projection." With the keywords "xray" and "lateral" we might also find 
images that show the "lateral wall of the trachea" , but have a postero-anterior 
projection. In this case the query results have a low precision. We measured 
recall and precision in a preliminary study with the same data set and published 
the results in [S]. 

The requests r% - r§ describe relational information, e.g. a finding in a par- 
ticular location or a finding with a qualifier. Neither MA\ nor MA 2 and MA3, 
however, can capture relational information. For example, if there is an image 
annotation that contains a "soft tissue mass" located in "chest wall" and a 
"thickening" of the "pleural membrane" , it would be retrieved with any of the 
queries for MA\ - MA3, although it is not an answer to r^. Again, these queries 
lead to low precision. 

The request ^5 expresses a causal relationship between two findings, i.e. 
"metastases that derive from a carcinoma." Only MA5 is expressive enough 
to capture this information by using the derivingFrom property that was added 
to SNOMED CT for this modelling approach. 

The framework also helps us to compare the fitness of the modelling ap- 
proaches. MA\ — MA5 differ in expressivity, complexity and required modelling 
effort. MA\ is the least expressive, but also the least complex and involving com- 
paratively little modelling effort . The textual image descriptions do not have to 
be processed in any way and the queries are just natural language keywords, i.e. 
the user does not have to know a particular query language. However, as stated 
above, recall and precision of the query results are lower than in the more expres- 
sive modelling approaches because MA\ does not take into account synonyms, 
homonyms, relational information etc. 

The queries in MA2 are equally good as the ones for MA\ . This approach does 
not involve a formal query language (we just used the keywords from the En- 
glish representation of the respective information request which is then mapped 
to SNOMED CT terms automatically). Essentially, the user can use the same 



keywords as in MA\. Since MA 2 can recognise synonyms and taxonomical re- 
lationships (which are defined in the thesaurus), its retrieval results are much 
better than those of MA\. It has to be noted however, that the tools we used 
to automatically map the text to concepts of the SNOMED CT ontology only 
know a finite number of synonyms for each concept. Therefore, natural language 
queries might lead to losses in recall. Losses in precision are also possible if 
homonyms are mapped to the wrong concept. 

MA 3 is very similar to MA 2 in the sense that it merely lists SNOMED CT 
classes that were identified in the textual image descriptions. However, in MA3 
the mapping has been done manually and not with the support of tools as 
in MA 2 . The manual mapping involved a considerably higher data modelling 
effort. Additionally, for MA3 the queries are formulated as OWL class expres- 
sions, which are more complex than natural language keywords. In summary, 
the queries in MA2 are slightly fitter than the ones in MA3 and the quality of 
the results for both modelling approaches is comparable. 

AIA4 and MA§ on the other hand are very expressive and capture the mean- 
ing of the original image descriptions very well and therefore lead to better recall 
and precision of the query results compared to the other modelling approaches. 
But that comes to the cost of a much higher modelling effort (due to the manual 
translation and the modelling of relational information) and higher complexity 
of the queries. An interesting observation is that a little more modelling effort 
on the schema of AIA§ (i.e. adding some roles and role hierarchy axioms) greatly 
improves the queriability of this approach compared to the relatively similar 
approach MA4 without these axioms. Both in the image annotations and in the 
queries, cumbersome SNOMED CT specific constructs like roleGroups can be 
avoided and lead to a better readability of the annotations and fitter queries and 
therefore increased queriability. 

In general the evaluation framework helps us to understand: 

— which are the weak spots of each modelling approach (e.g. low quality of 
query results in MA±, high complexity of the queries in MA4) 

— which are the strong spots of each modelling approach (e.g. MA 5 captures 
the meaning particularly well and has high recall and precision) 

With this knowledge the system designer can now assess the trade-offs be- 
tween the advantages and disadvantages that these measurements indicate and 
the modelling effort that is required for each of the approaches. Based on this, 
informed decisions can be made. For example, that MA2 and MA3 are more or 
less similarly fit but MA3 requires significantly more effort due to the manual 
translation of the data. Therefore, MA2 might be preferred over MA3. 



Another observation could be that if the precision of the results is not crucial, 
one might prefer a simpler modelling approach (e.g. MA2) that involves less 
modelling effort than the more expressive approaches but still leads to acceptable 
results. However, if recall and precision of the results are of high priority, one 
might want to invest some more effort and select MA$. 

Furthermore, the evaluation makes salient how we could improve or customise 
a modelling approach in order to make it better or make use of its full potential. 
For example, modifying MA4 by adding some role axioms to the schema leads 
to a much fitter modelling approach MA5. 

5 Conclusion and Future Work 

We compared heterogeneous modelling approaches for an information system for 
medical image annotations. In order to evaluate the strengths and weaknesses 
of these approaches we used an information system evaluation framework that 
was designed to measure the fitness of modelling approaches with a particular 
focus on their qucriability. The evaluation framework can make interesting char- 
acteristics of a modelling approach salient and therefore help information system 
designers to assess the trade-off between modelling effort and qucriability and 
to make an informed decision about which modelling approach is most suitable 
for their needs. 

Currently we are refining the evaluation framework so that it can be used 
for a wider range of scenarios and applications. We are extending the framework 
to measure not only the fitness of exact queries, but also such that answer 
information requests only partially. Furthermore, we incorporate measurements 
of false positives and false negatives in the query answers in order to combine 
the fitness of a query with retrieval performance measurements. Other possible 
extensions are e.g. measuring the flexibility of a system by evaluating in how 
many ways a user can formulate a fit query for an information request. Another 
adornment would be to combine the fitness values with measurements for query 
answering performance so that more information about the system can be taken 
into account. We are currently evaluating these extensions in other case studies. 

The particular case study presented in this paper lead to the conclusion 
that ontology-based image annotations can lead to good retrieval performance 
in terms of recall and precision and good queriability if an appropriate modelling 
technique is used. However, it turned out that this involves high modelling effort 
since manual intervention in the translation process is required. It would be 
worth to investigate whether it is possible to combine MA2 and MA§ by using 
text mining tools not only to automatically map concepts but also relational 
information. Another important observation we made is that SNOMED CT is 
suitable for modelling image descriptions and - most important - that a rather 



small customisation, i.e. adding some role axioms in order to bypass cumbersome 
SNOMED CT constructs such as roleGroups, can lead to great benefits in terms 
of queriability and comprehension. 
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